Neural Information Processing Systems http://nips.cc/

We consider the minimization of submodular functions subject to ordering constraints.

In this paper the authors consider the problem of minimizing a continuous submodular function subject to ordering (isotonic) constraints. They first show that the problem can be solved if we first discretize it (per coordinate, not in [0,1] n), and then solve the resulting discrete optimization problem using convex optimization. The fact that the problem is solvable in polynomial time is of course not surprising, because, as pointed out by the authors in lines 29-36, we can add a penalty to the objective that will implicitly enforce the constraints. However, this can significantly increase the Lipschitz constant of the objective, and that is why the authors take on an alternative approach. First, they prove that seen in the space of measures, the isotonic constraints correspond to dominating inequalities of the CDFs, which I guess is an intuitive result given the results known for the unconstrained case.

The SVM kernel is a function which takes low dimensional input space and transforms it to a higher dimensional space. It converts not separable problem to separable problem. It is mostly useful in non-linear separation problem. It performs extremely complex data transformations, then finds out the process to separate the data based on the labels or outputs defined. To train a support vector classifier and then optimize the objective function, we must perform some operations with the higher dimensional vectors.

A tutorial on the sigmoid function, its properties, and its use as an activation function in neural networks to learn non-linear decision boundaries.

AI has a long history. One can argue it even started long before the term was first coined; mostly in stories and later in actual mechanical devices called automata. This chapter only covers events relevant to the periods of AI winters without being too exhaustive in hope to extract knowledge that can be applied today. To aid understanding the phenomenon of AI Winters, the events leading up to them are examined. Many early ideas about thinking machines appeared in the late 1940s to '50s by people like Turing or Von Neumann. Turing tried to frame the questions of "Can machines think?" differently and created the imitation game, now famously called the Turing Test.

We consider the minimization of submodular functions subject to ordering constraints. We show that this potentially non-convex optimization problem can be cast as a convex optimization problem on a space of uni-dimensional measures, with ordering constraints corresponding to first-order stochastic dominance. We propose new discretization schemes that lead to simple and efficient algorithms based on zero-th, first, or higher order oracles; these algorithms also lead to improvements without isotonic constraints. Finally, our experiments show that non-convex loss functions can be much more robust to outliers for isotonic regression, while still being solvable in polynomial time.

We consider the minimization of submodular functions subject to ordering constraints. We show that this potentially non-convex optimization problem can be cast as a convex optimization problem on a space of uni-dimensional measures, with ordering constraints corresponding to first-order stochastic dominance. We propose new discretization schemes that lead to simple and efficient algorithms based on zero-th, first, or higher order oracles; these algorithms also lead to improvements without isotonic constraints. Finally, our experiments show that non-convex loss functions can be much more robust to outliers for isotonic regression, while still being solvable in polynomial time.

I came across this paragraph about logistic regression with PCA in Kevin P Murphy's book on Machine Learning. If we use PCA first, then use logistic regression afterwards, although overall, this is still representable as a logistic regression problem, the problem is constrained since we have forced linear regression to work in the subspace spanned by the PCA vectors. Consider 100 training vectors randomly positioned in a 1000 dimensional space each with a random class 0 or 1. With very high probability, these 100 vectors will be linearly separable. Now project these vectors onto a 10 dimensional space: with very high probability, 100 vectors plotted in a 10 dimensional space will not be linearly separable.

Most of the beginners start by learning regression. It is simple to learn and use, but does that solve our purpose? Because, you can do so much more than just Regression! Think of machine learning algorithms as an armory packed with axes, sword, blades, bow, dagger etc. You have various tools, but you ought to learn to use them at the right time.